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I. REAL PARTY IN INTEREST 

The real party in interest is FANUC LTD, the assignee of the subject application. 
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II. RELATED APPEALS AND INTERFERENCES 

Appellant, appellant's legal representative, and the assignee do not know of any prior or 
pending appeals, interferences or judicial proceedings which may be related to, directly affect or 
be directly affected by, or have a bearing on, the Board's decision in this appeal. 
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STATUS OF CLAIMS 

Appealed claims 1-32 have been rejected and are on appeal. 
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IV. STATUS OF AMENDMENTS 

An Amendment After Final was filed concurrently with the Notice of Appeal in response 
to the final Office Action mailed March 31 , 2006. In the Amendment After Final, claim 33 was 
cancelled. As such, the Amendment After Final is believed to be in accordance with 37 CFR 
1.116 (b)(1) by only canceling a claim. 
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V. SUMMARY OF CLAIMED SUBJECT MATTER 

Independent claim 1 recites a robot system having an image processing function for 
determining orientation, or orientation and position of a robot operation on one of a plurality of 
objects W. 

Claim 1 further recites a robot (e.g. robot RB in Fig. 1) and a first image capturing device 
21 capturing image data of the plurality of objects W containing respective images of the objects 
W (Fig. 1, p. 5, II. 20-22). A memory 2 storing reference models (Fig. 3, p. 6, II. 25-26), each 
comprising an image of a reference object captured by the image capturing device 21 in a 
different direction (Figs. 2a-2d, p. 12, II. 9-11), and for each reference model storing information 
of the capturing direction of its associated image and information of an orientation of the robot 
RB with respect to the reference object (Fig. 5, p. 13, II. 14-19), the information of the capturing 
direction representing a rotational posture of the reference object relative to the robot, the 
reference object being one of the plurality of objects or an object having a shape identical to that 
of one of the plurality of objects W (p. 1 1 , II. 24-26). 

Claim 1 further recites a processor 31 (Fig. 4) to perform matching on the image data 
containing images of the plurality of objects captured by the first image capturing device (p. 11, 
II. 8-14) with each of the reference models successively to select one object having an image 
matched with one of the reference models, and to determine orientation, or orientation and 
position of the robot RB operation based on the image of the selected one object, based on the 
one reference model and the information of its associated capturing direction, and based on the 
information of the orientation of the robot RB operation with respect to the reference object that 
is associated with said one reference model (p. 8, II. 10-14). 

Independent claim 12 is directed to a robot system having an image processing function 
for determining orientation and recites features that are similar to claim 1 and which are 
supported by the specification in a similar manner. 

Independent claim 23 is directed to a method for automatically determining an 
arrangement of a workpiece relative to a robot performing a method as described above with 
respect to claim 1 . 

None of the claims contain an element expressed as a means or step for performing a 
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specified function without the recital of structure, material, or acts in support thereof. 
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VI. GROUNDS OF REJECTION TO BE REVIEWED ON APPEAL 

Claims 1 , 12 and 23 stand rejected under 35 U.S.C. §1 03(a) as being unpatentable over 
Spight (US 4,462,046) in view of Corby, Jr. et al. (US 5,745,387). 
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VII. ARGUMENT 

A. Review of the Prior Art 

L U.S. Patent No. 4,462,046 ("Spight") 

Spight discloses a machine vision system utilizing programmable optical parallel 
processing. The scene being viewed by the vision system of Spight has been monitored by a 
video camera and converted into an electrical signal which produces an optical image on a 
object reproduction display 10. Simultaneously, a reference display indicative of a desired object 
to be identified is displayed on a reference display 12 from data stored within a memory of the 
system control processor. Spight, 3:29-3:38. 

One embodiment of Spight would allow the incoherent optical to electrical converters 52 
to view a single perspective of the object Ob being viewed. Instead of iterating possible 
particular configurations of the object by viewing the object from various positions, a plurality of 
configurations of each desired object to be identified would be stored within the system control 
processor 64. For each iteration of the parallel optical processing of Spight, a new reference 
signal would be provided. Each desired object to be identified would be defined by a plurality of 
reference signals indicative of a number of particular orientations of the object being viewed. 
Thus, a plurality of iterations could be performed without the need to vary the apparent 
perspective of the scene viewed by the incoherent optical to electrical converters 52. 
Alternatively, the plurality of reference signals could be produced by computer generated rotation 
of translation images of each object to be identified. Spight, 4:1-4:19. 

The convolution of the object signal produced by the object display 10 and the reference 
signal produced by the reference display 12 is applied to a video detector 40 arranged at the 
focal distance of the inverse Fourier transform lens 36. This video detector may be a video 
camera or, alternatively, a charge coupled device (CCD) array. Thus, the object signal is 
correlated with the reference signal in a manner which is utilized by the remainder of the system 
as explained in conjunction with FIG. 2. Spight, 5:42-5:51. 

The objects Ob to be viewed by the vision system of Spight may be arranged randomly 
on a viewing table 50. Spight, 6:11-6:13. 

This cross-correlated Fourier transform signal .phi.(x,y)* IR(x,y) is applied to a correlated 
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signal optical to electrical converter 60 including video detector 40. The electrical correlation 
output of the correlated signal optical to electrical converter 60 is then processed by the system 
control processor 64 in a known manner in order to determine the degree of correlation between 
the object signal o(x,y) and reference signal r(x,y). When the degree of correlation between 
these respective signals exceeds a predetermined limit, the particular configuration then viewed 
by the incoherent optical to electrical converters 52 and their associated multiplexing interface 
54 is determined to be closely associated with the reference signal indicative of the desired 
object to be identified recalled from memory within the system control processor 64. A high 
degree of correlation informs the system control processor 64 that the scene being viewed is 
closely approximated by the computer reproduced reference scene. Thus, it is possible to both 
identify the objects being viewed Ob and determine their location and spatial orientation via the 
parallel optical correlation utilized in Spight. Spight, 7:28-7:49. 

The system of Spight, as controlled by the system control processor 64, thus repeatedly 
correlates a multiplicity of particular configurations of the object being viewed Ob with the 
reference signal indicative of a desired object to be identified to determine the correlation 
therebetween. If all configurations of orientation and position have been correlated with the 
reference signal and an insufficient correlation therebetween is discovered, the system control 
processor 64 may then recall a new data set indicative of a new desired object to be identified 
and thus a new reference signal for successive iterative comparison with particular 
configurations monitored by the incoherent optical to electrical converters 52 and their 
associated multiplexing interface 54. Thus, it is possible for the system of Spight to correlate a 
plurality of possible orientations of an object with a plurality of reference signals representative of 
desired objects to be identified in order to determine what object is being viewed and its position 
and orientation in space. Spight, 7:64-8:15. 

Once the object is identified, the system control processor 64 provides this information 
and object position information to an effector control processor 200 which may drive a robotic 
effector or manipulator 202 to perform any desired task on the object being viewed. The position 
of the object is determined on the basis of the position of the bright points 212 viewed by the 
video detector 40. Because of the use of parallel optical processing which facilitates rapid 
iterations of the correlation between particular configurations of the object being viewed and the 
reference signals indicative of the object to be identified, the system of Spight may identify the 
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object, its location and orientation in a substantially real-time manner. Thus, even objects Ob 
moving along a conveyor line may be detected by the vision system of Spight and identified with 
an understanding of the object's location and orientation in sufficient time to perform a 
manipulative operation on the object via robotic effector 202 before the object is shifted, to any 
great extent, by the conveyor. Spight, 8:16-8:37. 

This monitoring and evaluation of the correlation signal may be performed in order to 
evaluate the degree of correlation between the reference view information indicative of the 
desired object to be identified and the object or scene being viewed. If sufficient correlation 
exists (S13), the system control processor 64 in connection with the effector control processor 
200 control the robotic effector 202 based on the reference view information being considered 
(S15). If no sufficient correlation exists, then the system control processor may either increment 
the reference view information, or alternatively change the angle the object is being viewed from, 
by controlling the signals received from the incoherent optical to electrical converter 62 by the 
multiplexing interface 54 (S14). Depending upon the embodiment for which the system control 
processor is programmed, one of these parameters will be incremented in order to iteratively 
cross-correlate new combinations of reference view information and information indicative of 
objects being viewed. Thus, the correlation program is again initiated. Spight, 9:59-10:1 1 . 

If a high degree of correlation exists, the system control processor 64 provides the 
effector control processor 200 with information related to the identification, spatial orientation, 
and position of the object being viewed, this information being derived from the information 
related to the desired object to be identified or reference information produced within the system 
control processor. If a high degree of correlation is not obtained from the correlation of the 
object signal and reference signal, the system control processor either compares the same 
object signal with a new reference signal or compares the same reference signal with a new 
object signal in order to iteratively correlate information related to a desired object to be identified 
stored within the system control processor 64 with information related to the scene being viewed 
or reference object positioned on the viewing table 50. Spight, 1 1 :21-1 1 :37. 

ii. U.S. Patent No. 5,745,387 ("Corby") 

Corby, Jr. discusses an augmented reality maintenance system employing a manipulator 
arm with an archive and comparison device. Corby, Jr. discusses that the use of manipulator 
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arms typically requires a method of determining the position and orientation of the distal end of 
the manipulator arm (and/or its subparts) with respect to the operating environment. This is 
required to successfully move the manipulator arm through the inspection environment without 
colliding with structures in the environment. Manipulator arms can be used to determine 
physical changes in an environment. One problem arises is that irregularities need to be 
monitored over a period of time (on the order of years) to determine the rate of deterioration. 
Corby, Jr. states that this is accomplished by moving the manipulator arm to a particular position 
and videotaping the structure or device which is to be examined. At a later date the manipulator 
arm is positioned at the same site and current data (such as a video image) is compared to 
previous data. Since it is very difficult to obtain sensor data with the same parameters, it is 
difficult to determine differences in the structure over time. This tends to be a hit-or-miss type of 
adjustment with a very subjective determination being made by the operator. Corby, Jr., 1 :44- 
1:64. 

B. Claims 1, 12 and 23 are patentable over U.S. 4,462,046 ("Spight") in view of U.S. 
Patent No. 5,745,387 ("Corby") 

In the final Office Action, the Examiner rejected independent claims 1,12 and 23 as 
being unpatentable over Spight in view of Corby. As these claims stand or fall together, the 
Appellant's argument is focused solely on the rejection of claim 1 . 

i. Spight does not disclose (1) the same image capturing device capturing 
both image data of objects and images of reference models or (2) storing a 
capturing direction of the reference image and information of an orientation 
of the robot with respect to the reference object 

Claim 1 recites: "a first image capturing device capturing image data of the plurality of 

objects containing respective images of the objects." Claim 1 also recites "a memory storing 

reference models, each comprising an image of a reference object captured by said image 

capturing device in a different direction". Because the same image capture device has been 

used, matching and orientation determination may become simple and accurate. 
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The Spight reference does not provide any explanation of how to derive its configuration 
data (in memory) used go generate reference signals. A fundamental feature of Spight is that 
"system control processor 64 recalls from its memory data indicative of a reference object or 
desired object to be identified." Spight, 6:41-6:44. Spight also discloses, "the scene being 
viewed is closely approximated by the computer reproduced reference scene." Spight, 7:44- 
7:46. Further, in Spight, "the system control processor 64 recalls reference view information 
from memory located in the system control processor 64." Spight, 9:29-9:31. Spight stores data 
in memory that can be converted to an optical signal for optical correlation processing. 
However, Spight does not disclose or suggest how the reference data is obtained. The only 
teaching in Spight is that "the plurality of reference of signals could be produced by computer 
generated rotation of translation images of each object to be identified." Spight, 9:17-9:19. 

Spight discloses only two teachings about reference data in computer memory. 

First, Spight generally discloses, "a plurality of configurations of each desired object 
would be stored in the system control processor 64." Spight, 9:6-9:8. Each configuration would 
be used to generate a corresponding reference signal. Spight, 9:10-9:13. However, in this first 
teaching Spight is completely silent on (1) how the configurations are obtained, and on (2) 
storing a capturing direction of the reference image and information of an orientation of the robot 
with respect to the reference object. 

The Appellant respectfully submits that Spight does not disclose or suggest using the 
same capture device for both capturing reference images and for capturing an image of the 
object whose identify and location is to be determined. 

The Appellant further submits that Spight does not disclose storing "information of the 
capturing direction of its associated image and information of an orientation of the robot with 
respect to the reference object," which "represents a rotational posture of the reference object 
relative to the robot" as is recited in claim 1 . 

Second, Spight discloses only one technique for deriving the configuration data or 
reference data stored in memory (the source of the reference signals). In Spight, the reference 
signals are produced by "computer generated rotation of the translation images of each object to 
be identified" (e.g. CAD-generated data). Spight, 9:16-9:19. This clearly does not meet the 
recitations of claim 1 mentioned above. Furthermore, this use of computer-generated reference 
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data (e.g. CAD-generated data) suggests that Spight can operate without using information of 
the capturing direction of an associated reference image and information of an orientation of the 
robot with respect to the reference object relative to the robot. Spight does not disclose details 
of reference models/images as discussed above with reference to claim 1 . 

Claim 12 recites features similar to claim 1 . 

Claim 23 recites "storing reference images corresponding to images of the workpiece or 
an object so shaped (workpiece/object) and reference arrangement information indicating 
arrangements of the robot and workpiece/object relative to each other when the images were 
captured, the reference arrangements comprising rotational arrangements of the workpiece 
relative to the robot". As discussed above, Spight does not discuss or suggest how 
configurations are derived. Spight only mentions that the configurations are stored without 
explaining what a configuration entails or how it is obtained. Again, the reference must disclose 
the same level of detail as recited in the claim. 

ii. Spight does not disclose determining the orientation of the robot 
operation 

Claim 1 recites determining orientation of a robot operation based on the image of the 
object, the reference model, etc. In contrast, nowhere does Spight disclose or suggest 
determining the actual orientation of a robot operation. Spight discloses: 

Once the object is identified, the system control processor 64 
provides this [identity] information and object position information 
to an effector control processor 200 which may drive a robotic 
effector or manipulator 202 to perform any desired task on the 
object being viewed. Position of the object is determined on the 
basis of the position of the bright points 21 2 viewed by the video 
detector 40. Because of the use of parallel optical processing 
which facilitates rapid iterations of the correlation between 
particular configurations of the object being viewed and the 
reference signals indicative of the object to be identified, the 
system of the present invention may identify the object, its location 
and orientation in an extremely fast, substantially real-time manner. 
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Thus, even objects Ob moving along a conveyor line may be 
detected by the vision system of the present invention and 
identified with an understanding of the object's location and 
orientation in sufficient time to perform a manipulative operation on 
the object via robotic effector 202 before the object is shifted, to 
any great extent, by the conveyor. Spight, 8:16-8:37. 

Spight does not disclose providing object orientation information to the effector or 
manipulator 202. Even if Spight did provide this information, it has no teaching or suggestion of 
determining the orientation of a robot operation. There is no indication that orientation can even 
be used. Spight may use robot graspers such as suction cups or other types of graspers that do 
not require orientation. Orientation information, if used at all, may be used to determine which 
type of robot hand to use, whether grasping is possible at all, whether the object needs to be 
agitated or otherwise moved for purpose of being grasped, and so on. As mentioned above, an 
anticipatory reference must disclose the same level of detail as found in the claims. The claims 
recite determining the orientation (or orientation and position) of the robot operation. Spight 
does not disclose this detail and only explicitly discloses providing object position information to 
the effector control processor 200. 

iii. Spight lacks a processor to perform matching 

Claim 1 recites "a processor to perform matching on the image data ... with each of said 
reference models." The rejection under appeal compares the processor to the computer 64 in 
Spight. However, the computer 64 in Spight does not actually perform matching on image data 
with reference models. In Spight, correlation is performed by optical parallel processing. 
Correlation is by "an optical lens system" and "use of another optical lens in order to produce a 
cross-correlation of the viewed scene and a desired object". Spight, 2:53-3:1 . Spight does 
disclose a processor 64, but this processor does not perform matching. Rather, the processor 
64 evaluates the degree/level of optically-found correlation/matching as indicated by the optical 
correlation output. "The system control processor then monitors the degree of correlation..." 
Spight, 3:4-3:6. The processor 64 only evaluates the degree of correlation, which is not the 
same as performing matching itself. This difference is notable. 
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The type of optical system taught by Spight is expensive and inflexible. Furthermore, 
optical systems using lenses, splitters, and the like are not readily combined with digital systems. 
Therefore, one skilled in the art would find it difficult to combine Spight with non-optical systems. 

iv. Corby does not disclose that one image capturing device captures 
reference signals and image data of the object 

In Corby, it appears that the position of the manipulator arm is manipulated by an 

operator, and that the data of an object is captured. Corby merely discloses superimposing an 

image of the manipulator arm on an image of the environment so as to display the position of the 

manipulator arm relative to the environment on the monitor as viewed from a user-selected 

viewpoint. Corby does not discuss that reference data of the object is ever captured. 

v. Corby does not describe and enable the claimed invention with sufficient 
clarity and detail to establish that the subject matter already existed in the 
prior art and that its existence was recognized by persons of ordinary skill 
in the field of the invention 

Although Corby discusses that the use of a manipulator arm requires a method of 

determining the position and orientation of the distal end of the manipulator arm, Corby does not 

discuss how this is enabled. In the Office Action, only the background section of Corby, Jr. is 

cited, which does not provide sufficient detail to qualify as prior art for an obviousness rejection. 

C. Conclusion 

In summary, the Appellant submits that claims 1-32 patentably distinguish over the prior 
art. Reversal of the Examiner's rejection is respectfully requested. 

Respectfully submitted, 

STAAS & HALSEY LLP 



Date: Auuy Z1,2Q0l> By: ff/ttyLu i&Jip*ri 

3 Gregor^ W. Harper ' 



Registration No. 55,248 



1201 New York Ave, N.W., Suite 700 
Washington, D.C. 20005 
Telephone: (202)434-1500 
Facsimile: (202)434-1501 
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VIII. CLAIMS APPENDIX 

1 . (PREVIOUSLY PRESENTED) A robot system having an image processing 
function for determining orientation, or orientation and position of a robot operation on one of a 
plurality of objects, the system comprising: 

a robot; 

a first image capturing device capturing image data of the plurality of objects containing 
respective images of the objects; 

a memory storing reference models, each comprising an image of a reference object 
captured by said image capturing device in a different direction, and for each reference model 
storing information of the capturing direction of its associated image and information of an 
orientation of the robot with respect to the reference object, the information of the capturing 
direction representing a rotational posture of the reference object relative to the robot, said 
reference object being one of the plurality of objects or an object having a shape identical to that 
of one of the plurality of objects; and 

a processor to perform matching on the image data containing images of the plurality of 
objects captured by said first image capturing device with each of said reference models 
successively to select one object having an image matched with one of said reference models, 
and to determine orientation, or orientation and position of the robot operation based on the 
image of the selected one object, based on said one reference model and the information of its 
associated capturing direction, and based on the information of the orientation of the robot 
operation with respect to the reference object that is associated with said one reference model. 

2. (PREVIOUSLY PRESENTED) A robot system having an image processing 
function according to claim 1, wherein said reference models are obtained from a part of the 
image data of the reference object. 

3. (ORIGINAL) A robot system having an image processing function according to 
claim 1 , wherein said reference models are obtained by processing the image data of the 
reference object. 
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4. (ORIGINAL) A robot system having an image processing function according to 
claim 1 , wherein said first image capturing device comprises a camera for capturing two- 
dimensional image data, 

5. (ORIGINAL) A robot system having an image processing function according to 
claim 4, wherein said image data of the reference object are captured by said camera from a 
predetermined distance. 

6. (PREVIOUSLY PRESENTED) A robot system having an image processing 
function according to claim 1, further comprising: 

a second image capturing device; wherein 

said robot situates said second image data capturing device to have said determined 
orientation or to have said determined orientation and said determined position with respect to 
the selected one object, and wherein 

said processor processes second image data captured by said second image capturing 
device to detect position and/or rotational posture of the selected one object with respect to said 
second image data capturing device. 

7. (PREVIOUSLY PRESENTED) A robot system having an image processing 
function according to claim 1 further comprising: 

a second image capturing device for obtaining three-dimensional position; wherein 
said robot situates said second image data capturing device to have said determined 

orientation or to have said determined orientation and said determined position with respect to 

the selected one object, so that said second image data capturing device is directed to a 

characterizing portion of the object; and wherein 

said processor detects three-dimensional position and/or posture of the selected one 

object based on three-dimensional position of said characterizing portion obtained by said 

second image capturing device. 
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8. (PREVIOUSLY PRESENTED) A robot system having an image processing 
function according to claim 6, wherein said first image data capturing device is used as said 
second image data capturing device. 

9. (PREVIOUSLY PRESENTED) A robot system having an image processing 
function according to claim 6, wherein said second image capturing device comprises a three- 
dimensional visual sensor of spot-light scanning type capable of measuring distance between 
the sensor and an object. 

10. (PREVIOUSLY PRESENTED) A robot system having an image processing 
function according to claim 6, wherein said second image data capturing device comprises a 
structured-light unit for irradiating a structured light on the selected object and capturing an 
image of the object including the irradiated light on the object. 

11 . (PREVIOUSLY PRESENTED) A robot system having an image processing 
function according to claim 7, wherein said robot operation is an operation of picking up the 
selected one object from the plurality of objects, some of which are overlapped with each other. 

1 2. (PREVIOUSLY PRESENTED) A robot system having an image processing 
function for determining orientation, or orientation and position of a robot operation on one of a 
plurality of objects of plural kinds, where the orientation of the operation corresponds to a 
determined orientation of the one object, where the determined orientation is a rotational posture 
of the one object, the system comprising: 

a robot; 

a first image capturing device capturing image data of the plurality of objects containing 
respective images of the objects; 

a memory storing reference models, each comprising images of each of different kinds of 
reference objects corresponding to images captured by said first image capturing device, and 
storing indicia of the kinds respectively associated with said reference models, and information 
of a different orientation of the robot with respect to each of the different images of the reference 
object of each kind, where the captured information of orientation comprises information of a 
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rotational posture of the reference object relative to the robot, each of said reference objects 
being one of the kinds of the plurality of objects or having a shape identical thereto; and 

a processor to perform matching on the image data containing images of the plurality of 
objects captured by said first image capturing device with each of said reference models 
successfully to select one object having an image matched with one of said kinds of the 
reference models, and to determine orientation, or orientation and position of the robot 
operation, the determining based on the image of the selected one object, based on said one 
reference model, based on the indicia of the kind associated with said one reference model and 
the information of the orientation of the robot operation with respect to the reference object 
associated with said one reference model of said one kind. 

1 3. (PREVIOUSLY PRESENTED) A robot system having an image processing 
function according to claim 12, wherein said reference models are obtained from a part of the 
image data of the reference object. 

14. (ORIGINAL) A robot system having an image processing function according to 
claim 12, wherein said reference models are obtained by processing the image data of the 
reference object. 

15. (ORIGINAL) A robot system having an image processing function according to 
claim 12, wherein said first image capturing device comprises a camera for capturing two- 
dimensional image data. 

16. (ORIGINAL) A robot system having an image processing function according to 
claim 15, wherein said image data of the reference object are captured by said camera from a 
predetermined distance. 

1 7. (PREVIOUSLY PRESENTED) A robot system having an image processing 
function according to claim 12, further comprising a second image capturing device, wherein 
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said robot situates said second image data capturing device to have said determined 
orientation or to have said determined orientation and said determined position with respect to 
the object, and wherein 

said processor processes second image data captured by said second image capturing 
device to detect position and/or posture of the selected one object with respect to said second 
image data capturing device. 

1 8. (PREVIOUSLY PRESENTED) A robot system having an image processing 
function according to claim 12, further comprising: 

a second image capturing device for obtaining three-dimensional position; wherein 
said robot situates said second image data capturing device to have said determined 

orientation or to have said determined orientation and said determined position with respect to 

the selected one object, so that said second image data capturing device is directed to a 

characterizing portion of the object; and wherein 

said processor detects three-dimensional position and/or posture of the selected one 

object based on three-dimensional position of said characterizing portion obtained by said 

second image capturing device. 

19. (PREVIOUSLY PRESENTED) A robot system having an image processing 
function according to claim 17, wherein said first image data capturing device is used as said 
second image data capturing device. 

20. (PREVIOUSLY PRESENTED) A robot system having an image processing 
function according to claim 17, wherein said second image capturing device comprises a three- 
dimensional visual sensor of spot-light scanning type capable of measuring distance between 
the sensor and an object. 

21 . (PREVIOUSLY PRESENTED) A robot system having an image processing 
function according to claim 17, wherein said second image data capturing device comprises a 
structured-light unit for irradiating a structured light on the selected one object and capturing an 
image of the object including the irradiated light on the object. 



21 



Serial No. 09/546,392 

22. (PREVIOUSLY PRESENTED) A robot system having an image processing 
function according to claim 18, wherein said robot operation is an operation of picking up the 
selected one object from the plurality of objects, some of which are overlapped with each other. 

23. (PREVIOUSLY PRESENTED) A method for automatically determining an 
arrangement of a workpiece relative to a robot, where the determined arrangement comprises at 
least rotational posture arrangement of the workpiece relative to the robot, the method 
comprising: 

storing reference images corresponding to images of the workpiece or an object so 
shaped (workpiece/object) and reference arrangement information indicating arrangements of 
the robot and workpiece/object relative to each other when the images were captured, the 
reference arrangements comprising rotational arrangements of the workpiece relative to the 
robot; 

from a known arrangement of the robot, capturing a working image of the workpiece 
among a plurality of randomly arranged workpieces with an imaging device; 

finding one of the reference images that matches the workpiece in the working image; 

and 

determining an arrangement of the robot relative to the workpiece based on information 
indicating the known arrangement of the robot, and based on the reference arrangement 
information corresponding to the found reference image, where the determined arrangement 
comprises rotational arrangement of the workpiece relative to the robot. 

24. (PREVIOUSLY PRESENTED) A method according to claim 23, wherein reference 
images and reference arrangement information is obtained for workpieces/objects of different 
shapes, and wherein the finding comprises first determining that a reference image of one of the 
different shapes matches the working image of the workpiece, and then finding one reference 
image of the shape that best matches the working image. 
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25. (PREVIOUSLY PRESENTED) A method according to claim 23, wherein the robot 
is used to capture the reference images, and wherein the reference arrangement information 
represents arrangements of the robot when capturing the reference images. 

26. (PREVIOUSLY PRESENTED) A method according to claim 23, wherein a second 
imaging device is affixed to the robot and is used to determine additional arrangement 
information used to determine the known arrangement of the robot relative to the workpiece. 

27. (PREVIOUSLY PRESENTED) A robot system having an image processing 
function according to claim 7, wherein said first image data capturing device is used as said 
second image data capturing device. 

28. (PREVIOUSLY PRESENTED) A robot system having an image processing 
function according to claim 7, wherein said second image capturing device comprises a three- 
dimensional visual sensor of spot-light scanning type capable of measuring distance between 
the sensor and an object. 

29. (PREVIOUSLY PRESENTED) A robot system having an image processing 
function according to claim 7, wherein said second image data capturing device comprises a 
structured-light unit for irradiating a structured light on an object and capturing an image of the 
object including the irradiated light on the object. 

30. (PREVIOUSLY PRESENTED) A robot system having an image processing 
function according to claim 18, wherein said first image data capturing device is used as said 
second image data capturing device. 

31 . (PREVIOUSLY PRESENTED) A robot system having an image processing 
function according to claim 18, wherein said second image capturing device comprises a three- 
dimensional visual sensor of spot-light scanning type capable of measuring distance between 
the sensor and an object. 
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32. (PREVIOUSLY PRESENTED) A robot system having an image processing 
function according to claim 18, wherein said second image data capturing device comprises a 
structured-light unit for irradiating a structured light on an object and capturing an image of the 
object including the irradiated light on the object. 



33. (CANCELLED) 
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IX. EVIDENCE APPENDIX 

Not applicable 
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RELATED PROCEEDING APPENDIX 

Not applicable 
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